The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks

Overfitting is one of the fundamental challenges when training convolutional neural networks and usually identified by a diverging test loss. The underlying dynamics how flow activations induce overfitting however poorly understood. In this study we introduce perplexity-based sparsity definition to derive visualise layer-wise activation measures. These novel explainable AI strategies reveal surprising relationship between overfitting, namely an increase in feature extraction layers shortly before loss starts rising. This tendency preserved across network architectures reguralisation so that our measures can be used as reliable indicator for while decoupling network’s generalisation capabilities from its loss-based definition. Moreover, differentiable formulation explicitly penalise emergence during impact reduced on studied real-time. Applying penalty analysing well known regularisers common supports hypothesis effectively improve classification performance. line with other recent work topic, methods insights into contradicting concepts capacity demonstrating dense enable discriminative learning efficiently exploiting deep models without suffering even trained excessively.